Our choice of title may seem strange but we mean each word. In this talk, we are not going to be 
concerned with computations made “after the fact”, i.e. those for which data are available and 
which are being conducted for explanation and insight. 

Here we are interested in preventing S&C design problems by finding them through computation 
before data are available. For such a computation to have any credibility with those who absorb 
the risk, it is necessary to quantitatively PREDICT the quality of the computational results. 
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Please note two things: 

There are a large number of people at Langley Research Center who are working on these issues, 
but we got tasked with presenting this talk. 

We do not claim that these notions are original to us, but the application and emphasis may be. 
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We want to make two points here: 

1 . No answer or a qualitative answer to the question “How good is my answer?” is not good 
enough for assessing risk. We will address this point in more detail later 

2. Where insightful and accurate S&C predictions are most desperately needed is in the design 
environment. Making a computation after data have been obtained is not a prediction — it is 
an explanatory effort. Explanatory efforts can be very useful but they do not require 
prediction of uncertainty. Note that attempts at calibration do, however, require uncertainty 
assessments of both the prediction and the experimental data. Otherwise, one has no idea 
how “fuzzy” the calibration/validation is. 


This talk addresses the question: 

“How am I going to convince the 
risk taker that my computation is 
good enough?” 


[ for the particular environment of interest.] 
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Outline 


* Risk relates directly to the ability to quantitatively 
predict uncertainty 

* An S&C example and its interpretation based on 
point-of-view 

* Expanding the traditional quality question 

* A long-term strategy for creating a controllable 
process 

* A strategy for getting started 

- For getting useful results right now 

- For establishing the long-term process 


NASA Svmposium on COMSAC 


NASA Langley Research Center -3 


640 


This chart is designed to illustrate the relationship of uncertainty quantification to risk. 

At the left end of the chart, there is no defined and managed process in place and no uncertainty 
quantification is possible. For this state of affairs, the decision maker, i.e. the person or group 
that uses the computational results, necessarily assumes all of the risk associated with any 
inaccuracy of the prediction. 


At the other end of the chart, the computationalist predicts the uncertainty following protocols 
and certification procedures suitable for use in a Court of Law. For this state of affairs, the risk is 
assumed entirely by the computationalist and he or she can be sued. 



Why predictive uncertainty quantification (UQ) 



All Risk Assumed by Decision Maker None 
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The other stages progress from the left to right, but please note that even the very first stage 
beyond the state of no quantification requires definition of a process and some sort of 
management system for verifying that the process is being followed. 
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Most (or virtually all) CFD is performed today without quantifying the consequences of 
uncertainty to an outcome metric. When uncertainty has been considered, it is usually restricted 
to a limited assessment of grid effects; other sources (turbulence model, algorithm, parameters, 
user practices, ...) are generally left unaddressed. 

The chart contrasts two customer requirements for a CFD computation of pitching moment. On 
the left is a cruise transport trim application where the required accuracy of the Cm prediction is 
+/- 0.001. On the right is a high angle of attack S&C application where the performance 
requirement is to have at least -0.1 nose down authority. The scales are set accordingly for each 
customer requirement, and the chart also provides for some grid sensitivity information to be 
added. 


CFD without quantified uncertainty 


Contrasting Performance and S&C Quality Requirements 


Cruise trim predictions 

Required accuracy: +/- 0.001 
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1 realization (calculation) 
C m *- 0.1351 
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In the absence of quantified uncertainty, all that known is the deterministic result that Cm = - 
0.1351. It is not know to any level of confidence if this calculation meets either customer’s 
requirement. Under such circumstances, the prediction is of limited use. 
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This figure has the identical format to the previous one. However, the results from a fairly 
extensive uncertainty quantification process are included. Forty-five computations were 
performed at three different grid density levels. Simple statistics now tell us that Cm = -0.137 
+/- 0.017 at 95% confidence. This outcome includes a variety of uncertainty sources (different 
grids, different turbulence models, different flow solvers, etc.) 


The individual results are also shown on both the left and right sides of the figure to put them in 
the context of the two customer requirements. It is clear that the variation is (1) completely 
unacceptable to the cruise trim requirement on the left and (2) completely acceptable to the high- 
a S&C objective on the right. 



CFD with quantified uncertainty 


Cruise trim requirement: b- +/- 0.001 
Grid refinement didn’t help 
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Grid refinement didn’t matter 
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C m = - 0,137 +/■ 0.017 (+ 1 - 2-sigma coverage) 
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Simply put, uncertainty quantification entails determination of conventional terms (average, 
standard deviation, and confidence) subject to certain process requirements. However, practical 
techniques will be required to quantify computational uncertainty within available resources and 
on a timescale consistent to project requirements. 
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Looking at individual pieces of the quality problem shines more light on them and actually 
recognizes that they each require different processes and ways of thinking. They end up being 
separate disciplines which develop on their own and co-evolve as well. 



Changing the typical quality question 


• Traditional Question: 

- How can I get the right answer? 


• New (Totally Separate) Questions: 


1. How good does my answer need to be? 

2. How do I find out how good my answer 
really is? 

3. How do I get my answer fast enough? 
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The question “How good does my answer have to be?” can only be answered by the customer of 
the computational results, i.e. the risk taker. Of course, the customer should always be informed 
of the likely quality of the process before he/she commissions the work. Furthermore, the 
resources provided by the customer can have a significant impact on the possible quality of the 
computational results. 

Unfortunately, the general absence of quantitative predictions of computational uncertainty has 
led to a typical customer demand of “Do the best you can.” However, recent efforts at several 
institutions to establish wind tunnel data quality assurance programs have encouraged some 
customers, most notably performance groups, to revisit their quality needs and to develop well- 

Question 1 



* How good does my answer need to be? 

(i.e. “What quantitative quality level is needed?”) 

- The customers, i.e. the user, of the computational 
predictions are responsible for determining their 
quantitative quality requirements. 

- Ideally, they would develop a process for doing 
this on a consistent basis. 

- This has nothing to do with the computational 
process and is not the subject of this talk. 
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defined processes for establishing defendable uncertainty requirements. These uncertainty 
requirements are usually traceable to some design or regulatory requirement that must be met for 
the airframe program to succeed. Some of these requirements are not even technical in nature, 
but nevertheless must be met. 
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Answering this question is the present focus of the computational uncertainty quantification 
work at Langley Research Center. It is impossible in this short talk to address anything more 
than the general notions. We recommend the following references: 

P. J. Roache, “Verification and Validation in Computational Science and Engineering”, 
Hermosa, 1998. 

W. L. Oberkampf, T. G. Trucano, “Verification and Validation in Computational Fluid 
Dynamics”, Progress in Aerospace Sciences, Vol. 38, No. 3, 2002, pp. 209-272. 

J. M. Luckring, M. J. Hemsch, J. H. Morrison, “Uncertainty in Computational Aerodynamics”, 
AIAA- 2003-0409, January 2003. 


Question 2 


* How do I find out how good my answer 
really is? 

(i.e. “How do I find out the actual quantitative quality 
level of my prediction?”) 

- We need to create a process that can be controlled 
and evaluated, 

- Which process should we use? Do we have to invent 
a new one? 

- What can we do right now to get started? 

- This is the main, albeit short, subject of this talk. 
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M. R. Mendenhall, R. Childs, “Best Practices for Reduction of Uncertainty in CFD Results”, 
AIAA- 2003-041 1, January 2003. 

M. J. Hemsch, “Statistical Analysis of CFD Solutions from the Drag Prediction Workshop”, 
AIAA- 2002-0842. 

M. J. Hemsch, “Statistical Analysis of CFD Solutions from 2nd Drag Prediction Workshop”, 
AIAA- 2004-0556, January 2004. 
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This question really addresses the issue of what process do I need? 

It is possible to use lower-order-physics codes for S&C problems as long as the domain of 
uncertainty predictability is known in advance. This means that the problem of interest would 
have to be pretty close to a previously-quantified domain. 

For true prediction, when such a previously-quantified domain does not exist, quantified 
uncertainty prediction does not seem possible. 

Note that this notion is especially important when novel configurations are being considered. 

Question 3 



* How do I get my answer fast enough? 

(i.e. “How do i get my predictive answer and its uncertainty 
fast enough?”) 

- This is not the subject of this talk, but we can’t resist a 
few comments. 

- Gotta have sufficient physics (or somehow put limit 
switches on reduced physics codes) 

- Gotta get designers to use them 
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We recommend the following reference for further reading on process quality assurance: 

M. C. Paulk, et al, “Capability Maturity Model for Software, Version 1.1”, Technical Report 
CMU/SEI-93-TR-024 (also ESC-TR-93-177), Software Engineering Institute, Carnegie Mellon 
University, February 1993 (download from SEI website). 

The above referenced document shows how to create and manage such a process. Paulk, et al 
have applied the approach to the software development process, but it applies just as well to any 
process, including uncertainty prediction/quantification. 


The strategy for the long-term is 


• Create a process that 
can be 

- Controlled 

- Evaluated 

- Improved 

(i.e. create a predictable process) 
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We would like to note that often the very act of measuring the outcome of a process (Evaluation) 
will lead to improvement in the process result. This was evident in the improvement of the 
Second Drag Prediction Workshop results over those of the First. We have also seen this in our 
development of statistical control of wind tunnel measurement processes. 
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If we think of prediction as a manufacturing process, then we have the situation described 
schematically above. We would never expect every widget coming off the line to have identical 
dimensions and, similarly, we should not expect every prediction to have no variation across, 
codes, grid types, users, turbulence models, etc. 

We do want to emphasize that to realize the full benefit of thinking this way and making it 
happen, it will be necessary to be fairly proficient at some basic statistical methods. The 
methods of greatest interest are the same ones used extensively in metrology and 
experimentation, particularly statistical quality/process control. Fortunately, these methods are 
not complicated. They do, however, require the user to get into a “statistical frame of mind” in 
order to use them effectively and correctly. 

Creating a predictable process ... 



Controllable input 
(variation that can be “fixed”) 



Predicted 
coefficients, 
flow features, 
etc. 


Uncontrolled input from the environment 
(variation that we have to live with, 
e.g. numerics, parameter uncertainty, 
model form uncertainty, users) 


[Danger! Danger! Statistics will have to be learned and used .] 
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In this presentation, we talk a lot about processes because the notion is fundamental to quality 
assurance, especially quantitative quality assurance. 

The best way that we know of to enable determination of quality is to think of computation as a 
process for manufacturing numbers. One of the biggest advantages of thinking this way is that 
we can borrow most of the methods and strategies of the manufacturing quality assurance 
community that have been developed over the last 80 years. In addition, we can borrow the 
extensions of those ideas to precision experimental work that have been developed at the 
National Bureau of Standards over the last 40 years. 


Critical levels of attainment for a predictable process 


• A defined set of steps 

• Stable and replicable 

• Measurable 

• Improvable 
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The quality assurance levels listed in the slide have been implicit in the quality literature but they 
were first promoted heavily by the Software Engineering Institute, (see reference on slide 11). 
These aspects are crucial for the credible prediction of computational uncertainty. The DoD 
actually has a process for certifying the quality assurance level attained on a sustained basis by a 
contractor’s software development process, ranging from Level 1 (no process) to Level 5 (all of 
the above attributes are included). 
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This breakdown of tasks was established by the National Bureau of Standard over 40 years ago 
for precision measurement in standards and calibration labs. It makes a seemingly impossible 
task not only tractable but controllable and credible. 

The first task, Calibration of Instruments, is done offline and provides a common reference state 
for all facilities which are traceable to national standards. 


The second task involves periodic offline testing of the measurement system using standard 
artifacts which are called check standards. This task is done solely for the purposes of tracking 
any possible drift in the mean or dispersion of the measurement output of the system. It also 
allows the credible characterization of that dispersion. 



How it’s done for an experimental process 



Testing in the 

presence of 
noise and error 
using a well- 
defined process 


Breakdown of tasks 


Noise is scatter in the process 
output due to minor variations 
in the way that the process is 
carried out. 

Error is bias in the process 
output due to modeling 
differences. 


Off-line 


Calibration of 
instruments 


Off-line 

Measuring the 
measurement system 

Off-line 
Discrimination testing 
of the measurement 
system 


QA checks against 
above measurements 
during customer 
testing 


Traceability to 
standards 

Random error 
characterization 
using standard 
artifacts 

Systematic error 
characterization 


Process output 
of interest 
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The third task involves the off-line determination of systematic errors in the measurement 
system. For a wind tunnel, some examples would be imperfect knowledge of the test section 
calibration coefficient and imperfect correction of wall effects. 

The fourth task involves those quality checks to be conducted during a test. Those checks are 
conducted by comparing data taken during the test for that purpose against historical data. There 
are many such checks that need to be done in a wind tunnel test. 
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The first task again provides referenceability by being able to prove that the code is doing what it is 
purported to do. This task is usually called “Code Verification” (Roache) 

The second task requires that the output variation of the computational process be controlled and 
evaluated. This can be effected through best practices and comparing the results of multiple codes, grid 
types, turbulence models, users, etc. There is a belief that attempts at grid convergence will be helpful 
here with part of this variation but preliminary results are not encouraging. This task is part of what is 
usually called “Solution Verification” (Roache). 


The third task involves parameter and model form uncertainty. There are a variety of ways to propagate 
parameter uncertainty into the code output and we are encouraged that these methods not only work but 

How it could be done for a computational process 




Computation in 

the presence of 
noise and error 
using a well- 
defined process 


Breakdown of tasks 


Off-line 


Verifying that the 
coding is correct 


Noise is scatter in the process 
output due to minor variations 
in the way that the process is 
carried out. 

Error is bias in the process 
output due to modeling 
differences. 


Off-line 


Measuring the 
computational process 


Off-line 


Model-to-model and 

model-to-reality 

discrimination 


QA checks against 
above measurements 
during computation for 
customer 


Traceable 
operational 
definition of the 
process 

Characterization 
of process 
variation 


Systematic error 
characterization 


Solution 

verification 
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can be reasonably implemented. Model form uncertainty is another story and much work needs to be done 
here. The most promising notion that we’ve seen is the idea from statistics of “severe testing” in which 
one attempts to find both the portions of the envelope where the predictions are reliable and the accuracy 
can be evaluated and the boundaries of those portions where the predictions become less reliable and 
accuracy becomes more difficult to predict. This task is usually called “Validation” (Roache). 

The fourth task involves checks to be made when a prediction is being made for a customer. Here it will 
be necessary (1) to assure that the best practice system is being followed so that predictions of process 
uncertainty have credibility and (2) to estimate the locations of the envelope boundaries where the 
credibility of the predicted systematic uncertainty becomes more problematical. This task is the on-line 
part of Solution Verification. 
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See http://aaac.larc.nasa.gov/tsab/cfdlarc/aiaa-dpw/. 



Recommended actions for COMSAC near-term 


* Establish a working group like the A1AA Drag Prediction 
Workshop (DPW) 

- Steer activities that can be started right now 

- Select a small number of COMSAC focus problems 

- Use those problems 

» to demonstrate the prediction uncertainty strategies 
we’ve proposed 

» to find out just how tough this problem really is 

* Some useful things to do right now for estimating 
uncertainty 

- Run multiple codes, different grid types, multiple turbulence 
models, etc. 

- Stick to realistic S&C problems, i.e. work data sets that fully 
capture the physics of the problem of interest. 
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Recommended actions for COMSAC far-term 


* Some key things to do right now to develop a powerful and 
reliable uncertainty quantification (UQ) process 

- Help us establish a reasonable UQ process. 

» Help us develop best practices and find ways to control and 
evaluate them. 

» Help develop and implement tools for propagating parameter 
uncertainty and IC/BC uncertainty into the coefficients of 
interest. 

» Help develop experiments to determine our ability to predict 
uncertainty and to predict the domain boundaries where the 
physics changes (and, therefore, probably the uncertainty). 
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We do not want, with the emphasis of this slide, to inadvertently give the impression that only 
on-line work counts. To the contrary, Slide 15 shows that we consider the off-line work 
described therein to be essential for a tractable and accurate process. By “local”, we simply mean 
local in the physical inference space (right physics). 



Final (can’t resist) slide 


• Validation for aerodynamics is not a 
global process 

- It is a never-ending local process and 
depends every time all the time on the flow 
you are working. 

- We can greatly improve, speed up and 
generalize (somewhat) the process we’re 
successfully using right now. 

- The key is quantitatively predicting the 
error. 
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